STAT 456: Introduction to Statistical Theory

MAIN THEOREM — Properties under Normality

Key observations:

• Part (a) is surprising! The sum and spread of a normal sample are independent.
• Part (b) follows from properties of normal (sum of normals is normal)
• Part (c) requires proof — we’ll do this in detail

PROOF of Independence (Part a)

Goal: Show \(\bar{X}\) and \(S^2\) are independent.

Strategy: Show \(\bar{X}\) and \((X_2 - \bar{X}, \ldots, X_n - \bar{X})\) are independent, then note \(S^2\) is a function of the latter.

Step 1: Write \(S^2\) in terms of deviations

\[S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar{X})^2 = \frac{1}{n-1}\sum_{i=2}^n (X_i - \bar{X})^2 + \frac{1}{n-1}(X_1 - \bar{X})^2\]

Since \(\sum_{i=1}^n (X_i - \bar{X}) = 0\), we have: \[X_1 - \bar{X} = -\sum_{i=2}^n (X_i - \bar{X})\]

So \(S^2\) is a function of \((X_2 - \bar{X}, \ldots, X_n - \bar{X})\) only.

Step 2: Transform to show independence

WLOG assume \(\mu = 0\), \(\sigma = 1\). Joint pdf of \((X_1, \ldots, X_n)\): \[f(x_1, \ldots, x_n) = \frac{1}{(2\pi)^{n/2}} \exp\left(-\frac{1}{2}\sum_{i=1}^n x_i^2\right)\]

Step 3: Make the transformation

\[y_1 = \bar{x}, \quad y_i = x_i - \bar{x} \text{ for } i = 2, \ldots, n\]

This is linear with Jacobian \(= 1/n\).

Inverse transformation: \[x_1 = y_1 - \sum_{i=2}^n y_i, \quad x_i = y_i + y_1 \text{ for } i \geq 2\]

Step 4: Compute joint density of \((Y_1, \ldots, Y_n)\)

After substitution: \[\sum_{i=1}^n x_i^2 = \left(y_1 - \sum_{i=2}^n y_i\right)^2 + \sum_{i=2}^n (y_i + y_1)^2\]

Expanding and simplifying: \[= ny_1^2 + \sum_{i=2}^n y_i^2 + \left(\sum_{i=2}^n y_i\right)^2\]

Step 5: Observe factorization

\[f(y_1, \ldots, y_n) = \frac{n}{(2\pi)^{n/2}} \exp\left(-\frac{ny_1^2}{2}\right) \exp\left(-\frac{1}{2}\left[\sum_{i=2}^n y_i^2 + \left(\sum_{i=2}^n y_i\right)^2\right]\right)\]

This factors as:

\[= \underbrace{\sqrt{\frac{n}{2\pi}} e^{-ny_1^2/2}}_{\text{density of } Y_1 = \bar{X}} \times \underbrace{g(y_2, \ldots, y_n)}_{\text{joint density of } Y_2, \ldots, Y_n}\]

Since joint pdf factors: \(Y_1 = \bar{X}\) is independent of \((Y_2, \ldots, Y_n)\).

Therefore \(\bar{X}\) is independent of \(S^2\). \(\blacksquare\)

PROOF of Chi-Squared Distribution (Part c)

Goal: Show \((n-1)S^2/\sigma^2 \sim \chi^2_{n-1}\).

Strategy: Induction on \(n\).

Notation: Let \(\bar{X}_k\) and \(S_k^2\) denote sample mean and variance based on first \(k\) observations.

Key Identity: \[(n-1)S_n^2 = (n-2)S_{n-1}^2 + \frac{n-1}{n}(X_n - \bar{X}_{n-1})^2\]

Base case (\(n=2\)):

\[S_2^2 = \frac{1}{1}\left[(X_1 - \bar{X}_2)^2 + (X_2 - \bar{X}_2)^2\right] = \frac{(X_1 - X_2)^2}{2}\]

So \((n-1)S_2^2/\sigma^2 = (X_1 - X_2)^2/(2\sigma^2)\).

Since \(X_1 - X_2 \sim N(0, 2\sigma^2)\), we have: \[(X_1 - X_2)/(\sigma\sqrt{2}) \sim N(0,1)\]

Therefore: \[S_2^2/\sigma^2 = [(X_1-X_2)/(\sigma\sqrt{2})]^2 \sim \chi^2_1 \quad \checkmark\]

Inductive step:

Assume \((n-2)S_{n-1}^2/\sigma^2 \sim \chi^2_{n-2}\).

From the key identity: \[\frac{(n-1)S_n^2}{\sigma^2} = \frac{(n-2)S_{n-1}^2}{\sigma^2} + \frac{(X_n - \bar{X}_{n-1})^2}{\sigma^2 \cdot n/(n-1)}\]

Now \(X_n\) is independent of \(X_1, \ldots, X_{n-1}\), hence independent of \(\bar{X}_{n-1}\) and \(S_{n-1}^2\).

Computing the distribution of the second term:

\[X_n - \bar{X}_{n-1} \sim N\left(0, \sigma^2 + \frac{\sigma^2}{n-1}\right) = N\left(0, \frac{\sigma^2 n}{n-1}\right)\]

So: \[\frac{(X_n - \bar{X}_{n-1})^2}{\sigma^2 n/(n-1)} \sim \chi^2_1\]

By independence and the additive property of chi-squared: \[\frac{(n-1)S_n^2}{\sigma^2} \sim \chi^2_{n-2} + \chi^2_1 = \chi^2_{n-1}\]

\(\blacksquare\)

Alternative Proof: The MGF/Covariance Approach

The alternative approach exploits a powerful property of the multivariate normal:

Alternative Proof of Part (a): Independence via Covariance

Goal: Show \(\bar{X} \perp S^2\) using the covariance approach.

Key insight: \(S^2\) is a function of the deviations \((X_1 - \bar{X}, \ldots, X_n - \bar{X})\).

If we show \(\bar{X}\) is uncorrelated with each \(X_i - \bar{X}\), and they’re jointly normal, then they’re independent.

The calculation:

\[\text{Cov}(\bar{X}, X_i - \bar{X}) = \text{Cov}(\bar{X}, X_i) - \text{Cov}(\bar{X}, \bar{X})\]

\[= \text{Cov}\left(\frac{1}{n}\sum_{j=1}^n X_j, X_i\right) - \text{Var}(\bar{X})\]

\[= \frac{1}{n}\text{Var}(X_i) - \frac{\sigma^2}{n} = \frac{\sigma^2}{n} - \frac{\sigma^2}{n} = 0\]

Conclusion: Since \(\bar{X}\) and \((X_1 - \bar{X}, \ldots, X_n - \bar{X})\) are:

1. Jointly normal (linear combinations of normals)
2. Uncorrelated (shown above)

They are independent. Since \(S^2\) is a function of the deviations, \(\bar{X} \perp S^2\). \(\blacksquare\)

Alternative Proof of Part (c): The Decomposition Identity

Goal: Show \((n-1)S^2/\sigma^2 \sim \chi^2_{n-1}\) using MGF ideas.

Step 1: Start with what we know

If \(Z_i = (X_i - \mu)/\sigma \sim N(0,1)\) iid, then: \[\sum_{i=1}^n Z_i^2 = \sum_{i=1}^n \left(\frac{X_i - \mu}{\sigma}\right)^2 \sim \chi^2_n\]

Step 2: The key algebraic identity

\[\sum_{i=1}^n (X_i - \mu)^2 = \sum_{i=1}^n (X_i - \bar{X})^2 + n(\bar{X} - \mu)^2\]

Proof of identity: Expand \((X_i - \mu) = (X_i - \bar{X}) + (\bar{X} - \mu)\) and use \(\sum(X_i - \bar{X}) = 0\).

Step 3: Divide by \(\sigma^2\)

\[\underbrace{\sum_{i=1}^n \left(\frac{X_i - \mu}{\sigma}\right)^2}_{\chi^2_n} = \underbrace{\frac{(n-1)S^2}{\sigma^2}}_{?} + \underbrace{\left(\frac{\bar{X} - \mu}{\sigma/\sqrt{n}}\right)^2}_{\chi^2_1}\]

Step 4: Use independence (from Part a) and MGF uniqueness

• The two terms on the right are independent (since \(\bar{X} \perp S^2\))
• If \(U + V = W\) where \(U \perp V\), then \(M_W(t) = M_U(t) \cdot M_V(t)\)
• We have: \(M_{\chi^2_n}(t) = M_{?}(t) \cdot M_{\chi^2_1}(t)\)

Solving: \((1-2t)^{-n/2} = M_{?}(t) \cdot (1-2t)^{-1/2}\)

Therefore: \(M_{?}(t) = (1-2t)^{-(n-1)/2}\)

This is the MGF of \(\chi^2_{n-1}\)! \(\blacksquare\)

Proof of Part (b): MGF Approach

Goal: Show \(\bar{X} \sim N(\mu, \sigma^2/n)\) using MGFs.

Direct calculation:

\[M_{\bar{X}}(t) = E\left[e^{t\bar{X}}\right] = E\left[e^{t \cdot \frac{1}{n}\sum_{i=1}^n X_i}\right]\]

\[= E\left[\prod_{i=1}^n e^{tX_i/n}\right] = \prod_{i=1}^n E\left[e^{tX_i/n}\right] \quad \text{(independence)}\]

\[= \prod_{i=1}^n M_{X_i}(t/n) = \left[M_X(t/n)\right]^n\]

Since \(X_i \sim N(\mu, \sigma^2)\), we have \(M_X(s) = e^{\mu s + \sigma^2 s^2/2}\).

\[M_{\bar{X}}(t) = \left[e^{\mu(t/n) + \sigma^2(t/n)^2/2}\right]^n = e^{\mu t + \sigma^2 t^2/(2n)}\]

This is the MGF of \(N(\mu, \sigma^2/n)\). By MGF uniqueness, \(\bar{X} \sim N(\mu, \sigma^2/n)\). \(\blacksquare\)

DERIVATION of \(t\) Distribution PDF

Goal: Derive the PDF of \(T = Z/\sqrt{V/p}\) where \(Z \sim N(0,1)\), \(V \sim \chi^2_p\), \(Z \perp V\).

Step 1: Joint PDF of \((Z, V)\)

Since \(Z\) and \(V\) are independent: \[f_{Z,V}(z, v) = f_Z(z) \cdot f_V(v)\]

\[= \frac{1}{\sqrt{2\pi}} e^{-z^2/2} \cdot \frac{1}{\Gamma(p/2) 2^{p/2}} v^{(p/2)-1} e^{-v/2}\]

for \(-\infty < z < \infty\) and \(v > 0\).

Step 2: Transformation

Define: \[t = \frac{z}{\sqrt{v/p}}, \quad w = v\]

This gives us: \[z = t\sqrt{w/p}, \quad v = w\]

Step 3: Compute the Jacobian

\[J = \begin{vmatrix} \frac{\partial z}{\partial t} & \frac{\partial z}{\partial w} \\ \frac{\partial v}{\partial t} & \frac{\partial v}{\partial w} \end{vmatrix} = \begin{vmatrix} \sqrt{w/p} & \frac{t}{2\sqrt{pw}} \\ 0 & 1 \end{vmatrix} = \sqrt{\frac{w}{p}}\]

Therefore: \(|J| = \sqrt{w/p}\)

Step 4: Joint PDF of \((T, W)\)

\[f_{T,W}(t, w) = f_{Z,V}(z(t,w), v(t,w)) \cdot |J|\]

\[= \frac{1}{\sqrt{2\pi}} e^{-t^2w/(2p)} \cdot \frac{1}{\Gamma(p/2) 2^{p/2}} w^{(p/2)-1} e^{-w/2} \cdot \sqrt{\frac{w}{p}}\]

\[= \frac{1}{\sqrt{2\pi p} \, \Gamma(p/2) 2^{p/2}} w^{p/2} e^{-w(1 + t^2/p)/2}\]

Step 5: Marginal PDF of \(T\) (integrate out \(W\))

\[f_T(t) = \int_0^\infty f_{T,W}(t, w) \, dw\]

\[= \frac{1}{\sqrt{2\pi p} \, \Gamma(p/2) 2^{p/2}} \int_0^\infty w^{p/2} e^{-w(1 + t^2/p)/2} \, dw\]

Step 6: Recognize the Gamma kernel

The integrand has the form of a \(\text{Gamma}\left(\frac{p+1}{2}, \frac{2}{1 + t^2/p}\right)\) density.

Recall that for Gamma\((\alpha, \beta)\): \[\int_0^\infty \frac{1}{\Gamma(\alpha)\beta^\alpha} x^{\alpha-1} e^{-x/\beta} \, dx = 1\]

So: \[\int_0^\infty w^{p/2} e^{-w(1 + t^2/p)/2} \, dw = \Gamma\left(\frac{p+1}{2}\right) \left(\frac{2}{1 + t^2/p}\right)^{(p+1)/2}\]

Step 7: Substitute back

\[f_T(t) = \frac{1}{\sqrt{2\pi p} \, \Gamma(p/2) 2^{p/2}} \cdot \Gamma\left(\frac{p+1}{2}\right) \left(\frac{2}{1 + t^2/p}\right)^{(p+1)/2}\]

\[= \frac{\Gamma\left(\frac{p+1}{2}\right)}{\Gamma(p/2) \sqrt{2\pi p} \, 2^{p/2}} \cdot \frac{2^{(p+1)/2}}{(1 + t^2/p)^{(p+1)/2}}\]

\[= \frac{\Gamma\left(\frac{p+1}{2}\right)}{\Gamma(p/2) \sqrt{\pi p}} \cdot \frac{1}{(1 + t^2/p)^{(p+1)/2}}\]

\[= \frac{\Gamma\left(\frac{p+1}{2}\right)}{\Gamma(p/2) \sqrt{p\pi}} \left(1 + \frac{t^2}{p}\right)^{-(p+1)/2}\]

for \(-\infty < t < \infty\). \(\blacksquare\)